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Abstract 

Conducting  a  Cognitive  Analysis  to  adequately  support  a  follow-on  design  effort  for 
innovative  decision  support  tools  is  a  tall  order  and  requires  specific  properties  to  exist  in 
the  CTA  in  order  to  be  successful.  This  paper  outlines  requirements  for  analytical 
methodologies  to  satisfy  this  need.  These  requirements  are  based  on  several  premises. 

The  first  premise  is  that  a  representation  shapes  cognition  and  collaboration,  which  guide 
behavior.  This  is  known  as  the  Representation  Effect  (Norman,  1993).  The 
representation  effect  summarizes  a  widespread  psychological  result  that  the  content  and 
context  of  a  problem  representation  (i.e.  its  reflection  in  the  design  of  user  interface)  can 
radically  alter  a  decision-maker’s  responses.  Thus,  how  a  command  and  control  system 
is  designed  influences  the  cognitive  work  that  needs  to  be  accomplished,  either 
improving  or  degrading  “sensemaking”  performance. 

The  second  premise  is  that  the  representation  must  reflect  the  essential  characteristics  of 
the  work  domain.  That  is,  the  relationship  (or  mapping)  between  the  visual  structure 
established  by  a  particular  representation  and  the  underlying  constraints  and  relationships 
within  the  work  domain  itself  is  fundamental  to  the  decision-maker’s  effectiveness  when 
using  the  visualization.  Without  an  explicit  specification  of  this  mapping,  it  is  impossible 
to  determine  if  the  visualizations  are  supporting  user’s  needs  as  intended,  or,  making  the 
“supported”  task  more  difficult  (Zhang,  1997).  Woods  (1991)  has  called  this  the 
‘Mapping  Principle.’ 

Taken  together,  these  two  premises  have  a  significant  impact  on  the  design  of  decision 
support  systems  (e.g.,  military  command  and  control  systems).  In  order  to  be  effective, 
C2  systems  must  be  based  on  a  cognitive  analysis  that  has  identified  the  rich  context  of 
decision-making  requirements  for  the  resulting  design  artifact.  Essentially,  an  analysis  of 
the  cognitive  work  must  identify  ‘what’  is  necessary  to  represent;  then  the  design  of  the 
interface  must  effectively  represent  that  information  for  accurate  perception/cognition  by 
the  user. 
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Overview 


Cognitive  Analysis  (CA)  encompasses  a  wide  variety  of  specific  techniques  with  similar 
diversity  in  their  goals.  From  our  perspective,  an  analytical-based  design  process 
requires  an  analysis  methodology  that  successfully  bridges  the  ‘gap’  between  field 
studies  (knowledge  elicitation)  and  system  development  (system  design  requirements). 
Such  a  CA  methodology  must  provide  a  framework  for  gathering,  analyzing  and 
structuring  the  resulting  knowledge  about  end-users’  information  requirements  and  for 
identifying  desired  system  functionality  -  particularly  of  system  components  responsible 
for  providing  decision  support.  Thus,  any  CA  approach  is  incomplete  unless  it 
incorporates  techniques  for  analyzing  the  operational  decision  making  environment  (i.e., 
the  context  in  which  a  support  system  will  be  embedded),  modeling  human  decision¬ 
making  (i.e.,  the  cognitive  demands  and  strategies  to  perform  within  the  operational 
environment),  defining  information  requirements  for  effective  decision-making, 
identifying  representation  requirements  to  satisfy  the  information  requirements,  and  most 
importantly,  explicitly  linking  to  the  follow-on  design  phase  that  generates  specific 
decision  support  concepts  based  on  the  analysis.  An  overview  of  this  approach  is 
illustrated  in  the  following  figure. 


I  nformation  Requirements: 

Defining  information  transfer 
requirements  for  effective 
decision-making 


Decision  Requirements: 

Defining  requirements  for  effective 
decision-making  in  work  domain 


Representation  Requirements: 

Mapping  domain  semantics  to  syntax  and 
dynamics  of  support  tools  to  achieve 
information  transfer  requirements 


Mapping  Principle 


CTA  Model: 

Modeling  Fundamental  Relationships 
and  Demands  in  the  Work  Domain 


Knowledge  Elicitation: 

Finding  demands,  decision-making 
challenges,  expert  strategies,  etc. 


Transition  to 
Design: 

Explicit  linking  to 
follow-on  design 
phase 


Figure  1.  A  framework  for  a  cognitive  analysis  process  as  the  basis  for  the  design  of  decision  support 

concepts. 


This  paper  discusses  the  insights  gained  from  the  design  of  multiple  decision  support 
systems  for  military  command  and  control  with  a  specific  emphasis  on  the  resulting 
criteria  for  cognitive  analysis  to  to  be  able  to  support  C2  decision  support  system  design. 
These  criteria  include: 
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•  Knowledge  elicitation  is  only  the  starting  point  -  cognitive  analysis  must  be 
based  on  knowledge  elicitation  techniques,  but  this  is  only  the  input  for  the 
analysis.  Information  from  knowledge  elicitation  must  be  transformed  into 
analytical  insights  to  serve  as  an  effective  basis  for  decision  support  system 
design. 

•  Capturing  the  way  the  world  works  -  cognitive  analysis  must  capture  and 
document  the  goals  to  be  achieved  in  the  domain  and  the  functional  means 
available  for  achieving  them.  This  includes  a  representation  of  the  functional 
concepts  and  their  relationships  that  serves  as  the  context  for  decision-making. 

The  most  challenging  aspect  of  this  task  is  the  discovery  of  the  most  abstract 
concepts  that  are  the  most  indicitative  of  expert  understanding  of  the  work 
domain. 

•  Building  a  set  of  complementary  analytic  artifacts  -  cognitive  analysis  must 
systematically  transform  the  problem  from  initial  insights  about  the  demands  of 
the  work  domain  to  support  concepts  for  those  demands.  There  needs  to  be  a 
linkage  between  these  artifacts  to  establish  a  sufficient  design  basis  for  the 
decision  support  concepts. 

•  Establishing  a  high  degree  of  correspondence-  cognitive  analysis  must  specify 
what  information  needs  to  be  represented  in  the  decision  support  system,  as  the 
mapping  from  work  domain  to  representation  design  to  extracted  information  is 
critical  to  the  effectiveness  of  the  decision  support  system.  This  mapping  must  be 
the  context  for  the  cognitive  analysis  effort. 

The  insights  and  lessons  learned  from  these  design  efforts  have  a  significant  impact  on 
design  efforts  where  the  objective  is  to  design  for  increased  “situation  awareness”.  Only 
by  following  a  structured  analysis  methodology  followed  by  insightful  representation 
design  techniques  can  the  mapping  from  work  domain  to  design  artifact  be  explicit  and 
have  any  hope  of  supporting  effective  decision  making  and  situational  awareness. 

Criteria  for  Cognitive  Analysis  to  be  able  to  Support  C2 
Decision  Support  System  Design 

#1:  Cognitive  Analysis  must  be  far  more  than  Knowledge  Elicitation. 

The  objective  of  Knowledge  Elicitation  (KE)  is  to  gather,  through  a  combination  of 
methods,  a  complementary  set  of  information  about  the  decision-making  problem  space 
under  consideration.  Typically,  a  CA  is  performed  based  on  a  variety  of  KE  activities 
and  types.  This  involves  interactions  with  expert  practitioners  in  the  domain  and  includes 
face-to-face  interviews  with  the  experts,  watching  the  experts  work  in  the  domain,  and 
verbal  protocol  techniques.  In  practice,  this  is  an  iterative,  progressively  deepening 
process. 

The  phrase  ‘bootstrapping  process’  has  been  used  to  describe  this  process  and  emphasizes 
the  fact  that  the  process  builds  on  itself  (Potter,  et  ah,  2000).  Each  step  taken  expands  the 
base  of  knowledge  providing  opportunity  to  take  the  next  step.  Making  progress  on  one 
line  of  inquiry  (understanding  one  aspect  of  the  field  of  practice)  inspires  progress  on 
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another.  One  starts  from  an  initial  base  of  knowledge  regarding  the  domain  and  how 
practitioners  function  within  it  (often  very  limited).  One  then  uses  a  number  of  KE 
techniques  to  expand  on  and  enrich  the  base  understanding  and  to  evolve  a  CA  model 
from  which  ideas  for  improved  support  can  be  generated.  For  example,  one  might  start 
by  reading  available  documents  that  provide  background  on  the  field  of  practice  (e.g., 
training  manuals,  procedures).  The  knowledge  gained  will  raise  new  questions  or 
hypotheses  to  pursue  that  can  then  be  addressed  in  interviews  with  domain  experts  and 
also  provide  the  background  for  interpreting  what  the  experts  say.  In  turn,  the  results  of 
interviews  or  exercises  may  point  to  complicating  factors  in  the  domain  that  need  to  be 
modeled  in  more  detail.  This  provides  the  necessary  background  to  create  scenarios  to  be 
used  to  observe  practitioner  performance  under  simulated  conditions  or  to  look  for 
confirming  example  cases  or  interpret  observations  in  naturalistic  field  studies. 

One  key  element  of  the  Knowledge  Elicitation  is  that  it  is  performed  iteratively  with  the 
CA  effort.  As  interim  analytical  results  from  the  modeling  task  (converting  raw  KE  data 
to  CA  findings)  become  available,  they  are  used  as  material  for  further  elicitation.  The 
process  of  constructing  the  CA  artifacts  provides  requirements  for  the  information  needed 
to  enrich  the  model.  Ideally,  the  CA  model  provides  a  mechanism  for  integrating 
seemingly  disparate  sources  of  information  from  a  KE  process  into  a  unified  analysis  and 
design  framework.  Thus,  the  focus  of  KE  is  to,  in  an  iterative,  incremental  manner, 
provide  the  data  necessary  to  construct  the  set  of  CA  artifacts. 

Therefore,  it  is  important  to  maintain  a  clear  distinction  between  KE  and  CA.  Too  often 
CTA  researchers  talk  about  KE  as  if  it  is  CA.  However,  they  are  fundamentally  two 
separate  activities  with  different  methodologies.  Effective  KE  is  essential  to  obtain  the 
critically-needed  material  for  the  construction  of  the  set  of  CA  artifacts.  The  exact  same 
KE  data  can  be  transformed  in  different  ways  to  reveal  different  insights  into  the 
demands  of  the  work  domain  and  the  decision-making  of  the  practitioners. 

As  was  evident  in  Figure  1,  results  from  Knowledge  Elicitation  supports  the  entire  set  of 
CA  artifacts.  For  example,  during  an  observational  investigation,  one  may  see  evidence 
of  a  specific  information  requirement  for  a  particular  decision.  Or,  in  an  interview  the 
SME  may  verbalize  an  especially  difficult  problem  that  may  provide  insights  into 
fundamental  demands  in  the  work  domain. 

#2:  Cognitive  Analysis  must  capture  the  fundamentals  of  the  work  domain  and 
resulting  decision-making. 

In  performing  CA,  two  mutually  reinforcing  perspectives  need  to  be  considered.  One 
perspective  focuses  on  the  characteristics  of  the  domain  and  the  cognitive  demands  they 
impose.  The  focus  is  on  understanding  the  way  the  world  works  (the  fundamental, 
underlying  basis)  and  what  factors  contribute  to  making  practitioner  performance 
challenging.  Understanding  domain  characteristics  is  important  because  it  provides  a 
framework  for  interpreting  the  second  perspective  -  practitioner  decision-making  (Why 
do  experts  utilize  the  strategies  they  do?  What  complexities  in  the  domain  are  they 
responding  to?  Why  do  inexperienced  practitioners  perform  less  well?  What  constraints 
in  the  domain  are  they  less  sensitive  to?)  and  because  it  helps  define  the  requirements  for 
effective  support  (what  aspects  of  performance  could  use  support,  what  are  the  hard  cases 
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where  support  could  really  be  useful)  as  well  as  the  bounds  of  feasible  support  (what 
technologies  can  be  brought  to  bear  to  deal  with  the  complexities  inherent  in  the  domain, 
which  aspects  of  the  domain  tasks  are  amenable  to  support,  and  which  are  beyond  the 
capabilities  of  current  technologies). 

The  challenging  aspect  of  this  task  is  the  systematic  discovery  of  the  most  abstract  of  the 
concepts  (Rasmussen,  1986)  as  the  most  indicative  of  expert  understanding  of  the 
domain,  most  challenged  during  unexpected  situations,  and  least  likely  to  be  captured  in 
existing  documentation  or  decision  aids.  The  physical  aspects  of  the  domain  can  be 
viewed  and  touched  and  inspected,  the  abstract  functional  concepts  require 
understanding.  An  effective  CA  properly  discovers  these  essential,  elusive  abstract 
concepts  and  how  they  relate  to  each  other. 

The  second  perspective  focuses  on  how  practitioners  respond  to  the  demands  of  the 
domain.  Understanding  the  knowledge  and  strategies  that  expert  practitioners  have 
developed  in  response  to  domain  demands  provides  a  second  window  for  uncovering 
what  makes  the  world  hard  and  what  are  effective  strategies  for  dealing  with  domain 
demands.  These  strategies  can  be  captured  and  transmitted  directly  to  less  experienced 
practitioners  (e.g.,  through  training  systems)  or  they  can  provide  ideas  for  more  effective 
support  systems  that  would  eliminate  the  need  for  these  compensating  strategies. 

A  comprehensive  CA  needs  to  encompass  analysis  and  modeling  of  both  of  these 
perspectives.  The  focus  is  on  building  a  model  that  captures  the  cognitive  task  analyst’s 
evolving  understanding  of  demands  of  the  domain,  knowledge  and  strategies  of  domain 
practitioners,  and  how  existing  support  tools  influence  performance  that  can  be  used  to 
guide  specification  of  requirements  for  improved  performance. 

An  impact  of  this  issue  is  in  the  nature  of  the  knowledge  being  sought  and  in  the  nature  of 
the  analysis  of  the  raw  data  collected  resulting  in  design  progress.  Virtually  any  of  the 
Knowledge  Elicitation  practices  can  form  the  input  data,  the  true  value  comes  in 
resolving  it  against  the  analytical  representation  in  the  CA  model  and  the  resulting 
conversion  of  that  data  into  the  formalisms  of  the  model. 

This  issue  is  at  the  heart  of  a  distinction  within  the  CSE  community  with  respect  to  the 
differences  between  Cognitive  Task  Analysis  (CTA)  and  Cognitive  Work  Analysis 
(CWA).  Rasmussen  and  his  colleagues  (Rasmussen,  1986;  Rasmussen,  et  al.,  1994) 
developed  a  framework  for  CWA  that  addresses  the  issues  of  how  to  analyze  human 
work  in  order  to  derive  implications  for  designing  computer-based  decision  support 
systems  explicitly  to  support  adaptation  to  novelty  rather  than  to  support  current  task 
sequences.  This  is  the  essence  of  what  Woods  and  Hollnagel  (2005)  refer  to  as  resilience 
-  the  ability  to  anticipate  and  adapt  to  surprise  and  error.  This  includes  issues  such  as 
failure-sensitive  strategies,  exploring  outside  the  current  boundaries  or  priorities, 
overcoming  the  brittleness  of  automation,  and  maintaining  peripheral  awareness  to 
maintain  flexibility.  For  this  reason,  we  argue  that  any  cognitive  analysis  must  model  the 
fundamental,  unchanging  properties  of  the  work  domain  that  must  be  supported 
independent  of  any  specific  task  demands.  This  is  the  fundamental  analysis  issue  to 
achieve  resilience  in  the  resulting  Joint  Cognitive  System  (Woods  and  Hollnagel,  2005). 

In  addition,  the  problem  is  even  harder  in  the  case  of  designing  decision  support  tools  for 
revolutionary  systems  (in  which  there  is  no  currently  existing  system),  CA  must  deal  with 
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the  problem  referred  to  as  the  envisioned  world  problem  (Woods,  1998).  That  is,  the 
introduction  of  any  computerized  support  system  will  change  the  nature  of  the  users’ 
tasks  (i.e.,  make  some  (hopefully  many)  easier,  some  (hopefully  few)  more  difficult, 
eliminate  some,  and  introduce  others).  The  envisioned  world  problem  means  that  CA 
faces  a  challenge  of  prediction  -  i.e.,  how  will  the  new  decision  support  tools  shape 
cognition  and  collaboration?  In  the  envisioned  world  problem,  we  cannot  simply  rely  on 
the  expertise  of  subject  matter  experts  of  the  current  system.  We  must  have  some 
mechanism  for  defining  the  fundamental,  unchanging  properties  of  the  work  domain  that 
will  need  to  be  supported  in  the  envisioned  world. 

The  envisioned  world  problem  is  the  basis  for  the  argument  by  Potter,  et  al.,  (2000)  for 
CTA  processes  to  extend  into  the  prototype  design  phase.  In  this  way,  understanding  of 
the  demands  of  the  work  environment  is  enriched  by  the  analysis  of  how  the  envisioned 
tools  change  the  cognitive  tasks  faced  by  the  users. 

In  Figure  1  we  include  a  Functional  Abstraction  Network  (FAN;  Elm,  et  al.,  2003)  as  a 
representative  CA  model  to  capture  the  fundamentals  of  the  work  domain.  While  there 
are  several  related  methodologies  (including  function-based  CTA  (Roth  &  Mumaw, 
1995),  goal-means  decomposition  (Woods  and  Hollnagel,  1987),  multi-level  flow 
modeling  (Lind,  1993),  cognitive  work  analysis  (Vicente,  1999),  cognitive  systems 
analysis  (Flach,  1998)  and  cognitive  function  analysis  (Boy,  1997))  we  advocate  the 
Applied  Cognitive  Work  Analysis  (ACWA;  Elm,  et  al.,  2003;  Potter,  et  al.,  2003).  This 
methodology  creates  a  function-based  goal-means  decomposition  that  models  the  goals  to 
be  achieved  in  the  domain  and  the  means  available  for  achieving  them.  The  critical 
benefit  of  this  approach  is  the  systematic  discovery  of  the  most  abstract  of  the  concepts 
(through  the  construction  of  a  FAN)  as  the  ones  most  indicative  of  expert  understanding 
of  the  domain,  most  essential  during  unexpected  situations,  and  least  likely  to  be  captured 
in  existing  decision  aids. 

A  FAN  is  a  recursive  goal-means  representation,  specifying  the  goals  to  be  accomplished, 
the  relationships  between  goals  (e.g.,  goal-sub-goal  relations,  mutually  constraining  or 
conflicting  goals),  and  the  means  available  to  achieve  the  goals  (e.g.,  alternative  methods 
available,  pre-conditions,  side-effects,  preferred  order)  at  ever  more  increasing  levels  of 
abstraction.  From  this  model  of  system  functioning,  it  is  possible  to  derive,  organize,  and 
preserve  the  goals,  processes,  critical  decisions,  and  information  requirements. 

The  rationale  for  this  approach  is  twofold.  First,  it  has  been  shown  that  an  explicit 
Functional  Abstraction  Network  model  closely  parallels  the  mental  models  of  some  of  the 
best  human  problem  solvers,  faced  with  high-stress,  high-value,  uncertain  (naturalistic) 
decision  making  conditions.  Second,  this  approach  has  been  shown  to  be  extremely 
valuable  in  designing  decision  support  tools  for  complex  processes  such  as  those  faced  by 
C2  teams  by  modeling  the  essential  underlying  concepts  and  relationships  in  the  work 
domain  that  need  to  be  an  integral  part  of  the  decision  support  system. 

The  resulting  FAN  is  independent  of  the  particular  agents  (human  or  machine)  required 
to  perform  the  work.  Rather,  it  specifies  the  specific  domain  objectives  and  the  functions 
that  must  be  available  and  satisfied  in  order  to  achieve  these  goals.  These  functions  may 
be  abstract  entities  that  need  to  have  other,  less  abstract  functions  available  and  satisfied 
in  order  that  they  might  be  achieved.  This  creates  a  decomposition  network  of  functional 
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objectives  or  purposes  that  are  linked  together  from  abstract  goals  to  specific  means  to 
achieve  these  goals.  The  intrinsic  work  tasks  that  are  present  in  the  work  domain  can  be 
mapped  to  the  goals  and  processes  identified  in  the  FAN. 

The  FAN  is  a  key  artifact  that  serves  as  the  initial  scaffolding  that  comprises  the  structure 
of  the  remainder  of  the  analysis  effort.  All  subsequent  phases  in  the  analysis  then  add 
depth  and  substance  onto  the  structure  (e.g.,  cognitive  work  and  associated  information 
requirements  overlaid  on  the  relevant  goal/process  relationships).  This  structure  defines 
the  shape  of  any  future  DSS  knowledge  model,  and  thereby,  the  shape,  nature,  and 
ultimately  the  content  and  form  of  the  communication  between  the  DSS  and  the  domain 
practitioners. 

#3:  Cognitive  Analysis  must  systematically  transform  knowledge  elicitation  into  a 
set  of  complementary  analytic  artifacts. 

In  order  to  achieve  this  criterion,  it  is  necessary  to  have  a  structured,  principled 
methodology  accompanied  by  a  set  of  analytical  artifacts  to  systematically  transform  the 
problem  from  initial  data  from  multiple  KE  sources  to  an  analysis  of  the  demands  of  a 
domain  to  design  basis  for  decision-aiding  concepts  that  will  provide  effective  support  for 
the  identified  demands. 

This  is  typically  accomplished  by  creating  a  set  of  artifacts  that  capture  the  results  of  each 
of  these  intermediate  stages  in  the  CA  process.  The  continuous  thread  formed  from  these 
artifacts  provides  a  principled,  traceable  link  from  KE  to  cognitive  analysis  to  design 
basis.  However,  the  progress  occurs  in  the  thought  and  work  in  accomplishing  each  step 
of  the  process;  by  the  process  of  generating  these  artifacts.  The  artifacts  serve  as  a 
mechanism  to  record  the  results  of  the  thinking  and  as  stepping  stones  for  the  subsequent 
step  of  the  process.  Each  intermediate  artifact  also  provides  an  opportunity  to  evaluate 
the  completeness  and  quality  of  the  analysis  effort,  enabling  modifications  to  be  made 
early  in  the  process.  The  linkage  between  artifacts  also  ensures  an  integrative  process; 
changes  in  one-artifact  cascades  along  the  thread  necessitating  changes  to  all  as  the 
process  is  repeated  in  several  expanding  spirals. 

As  was  evident  in  Figure  1,  there  should  be  a  linkage  between  the  complementary  set  of 
artifacts.  As  mentioned  earlier,  there  may  be  a  link  between  an  observation  during  a  KE 
event  and  an  information  requirement.  The  next  step  in  the  CA  process  is  the 
construction  of  links  between  the  CA  artifacts.  In  this  case,  it  would  need  to  be  links  in 
both  ‘directions’  -  backward  to  the  decision(s)  calling  for  that  information  requirement  as 
well  as  forward  to  the  representation  requirements  for  that  particular  information 
requirement. 

One  of  the  primary  impacts  of  this  issue  is  an  identification  of  the  critical  issues  in  the 
transition  from  CA  to  system  design.  This  includes  the  need  for: 

•  CA  to  go  well  beyond  analytical  insights.  A  CA  needs  to  provide  results  that  are 
able  to  be  transformed  to  design  artifacts,  preserving  linkage  at  an  atomic  level. 
For  example,  an  effective  CA  model  should  define  and  organize  the  information 
space  for  the  decision  support  system  (based  on  the  premise  that  the  information 
space  should  mimic  the  decision  space).  This  requires  a  CA  model  that  is  robust 
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enough  to  also  serve  as  a  design  model.  In  addition,  an  effective  CA  for  design 
needs  to  define  specific  information  requirements  (and  ideally,  representational 
requirements)  as  the  key  step  in  positioning  the  CA  as  a  design  basis. 

•  An  understanding  of  the  artifacts  used  by  system  designers  (e.g.,  system 
requirements)  and  how  results  from  a  CA  can  be  integrated  into  these  artifacts 
(and  effectively  support  system  design  activity).  Given  these  artifacts  form  the 
underlying  specification  for  system  development,  they  are  the  critical  targets  if 
CA  is  to  effectively  impact  design.  Therefore,  CA  results  need  to  be  in  a  set  of 
evolving  requirements. 

•  Artifacts  for  capturing  design  rationale  in  order  to  maintain  the  underlying  basis 
for  design  concepts  resulting  from  CA  effort  (in  order  to  separate  the  design  basis 
from  the  instantiation).  This  is  important  to  separate  the  information  requirements 
from  the  proposed  presentation  (in  order  to  isolate  the  source  of  the  problem  in  an 
ineffective  design). 

#4:  Cognitive  Analysis  must  serve  as  the  basis  for  innovative  decision  support 
system  design  concepts. 

The  use  of  cognitive  analysis  as  the  principal  component  of  defining  the  requirements  of 
cognitive  work  and  to  provide  a  foundation  for  design  of  new  decision-support  systems 
has  become  the  approach  for  a  significant  number  of  programs.  While  cognitive  analysis 
techniques  have  proved  successful  in  illuminating  the  sources  of  cognitive  complexity  in 
a  domain  of  practice  and  the  basis  of  human  expertise,  the  results  of  the  cognitive 
analysis  are  often  only  weakly  coupled  to  the  design  of  support  systems  (Potter,  Roth, 
Woods  and  Elm,  2000).  A  critical  gap  occurs  at  the  transition  from  cognitive  analysis  to 
system  design,  where  insights  gained  from  the  cognitive  analysis  must  be  translated  into 
design  requirements. 

In  order  to  avoid  this  critical  gap,  the  practice  of  CA  as  an  initial,  self-contained  work 
product  that  is  handed  off  to  system  designers  must  change.  In  fact,  Woods  (personal 
communication,  2004)  knows  of  only  one  CA  that  has  been  successfully  handed  off  to  an 
independent  DSS  development  team.  The  CA-based  design  practice  needs  to  evolve  into 
a  process  of  modeling  not  only  the  demands  of  the  domain,  the  strategies  and  knowledge 
of  practitioners,  but  also  the  informational  and  representational  requirements  to  support 
these  demands,  and  how  envisioned  artifacts  will  shape  these  strategies  and  coordinative 
activities. 

From  a  Cognitive  Systems  Engineering  perspective,  it  is  only  when  we  are  able  to  design 
appropriate  support  that  we  truly  understand  the  way  the  world  works  and  the  way  people 
will  operate  in  this  world.  This  is  the  essence  of  the  claim  by  Winograd  (1987)  that 
designing  “things  that  make  us  smart”  depends  on  “. .  .developing  a  theoretical  base  for 
creating  meaningful  artifacts  and  for  understanding  their  use  and  effects.”  This  is  also 
consistent  with  Tufte’s  (1997)  discussion  about  the  need  for  a  “causal  theory”  as  context 
for  assessing  information.  Therefore,  the  key  objective  of  a  CA  in  this  respect  is  a 
structured  representation  of  the  functional  concepts  and  their  relationships  to  serve  as  the 
context  for  the  support  system  to  be  designed.  This  structure  defines  the  shape,  nature, 
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and  ultimately  the  content  and  form  of  the  communication  between  the  decision  support 
system  and  the  domain  practitioner. 

The  relationship  between  the  visual  structure  established  by  a  particular  visualization  and 
the  constraints  and  relationships  of  the  domain  itself  is  fundamental  to  the  effectiveness 
of  the  visualization.  Without  an  explicit  specification  of  this  mapping,  it  is  impossible  to 
determine  if  the  visualizations  are  supporting  users’  needs  as  they  intended,  or  making 
the  “supported”  task  more  difficult.  Compared  to  the  research  that  has  been  done  on 
human  cognition,  very  little  work  has  been  done  describing  the  representational  nature  of 
visualizations  in  complex  information  processing  tasks  (Zhang,  1996). 

The  key  is  to  develop  a  mapping  between  information  on  the  behavioral  state  of  the 
domain  and  the  dynamic  syntax  of  the  visualization.  Woods  (1992)  has  called  this  the 
‘Mapping  Principle’  (Figure  2).  The  mapping  principle  means  that  one  cannot 
understand  information  displays  in  terms  of  purely  visual  characteristics,  but  that  it  also 
must  include  an  understanding  of  the  representational  requirements  of  the  work  domain. 
From  an  interface  design  perspective,  the  goal  is  to  reveal  the  critical  information 
requirements  and  constraints  of  the  decision  task  through  the  user  interface  in  such  a  way 
as  to  capitalize  on  the  characteristics  of  human  perception  and  cognition. 


Figure  2.  The  Mapping  Principle  (adapted  from  Woods,  1991)  dictates  that  design  of  a  decision 
support  system  must  be  based  on  the  informational  requirements  of  the  work  domain,  how  they  are 
reflected  in  the  representation,  and  ultimately  how  they  are  observed  by  the  user. 

There  are  two  fundamental  issues  involved  in  the  Mapping  Principle: 

•  Correspondence  -  an  issue  of  content.  What  information  should  be  present  in  the 
interface  in  order  to  meet  the  cognitive  demands  of  the  work  domain?  This  is  the 
mapping  from  the  work  domain  to  the  representation  of  the  problem  in  the 
visualization.  This  is  the  issue  that  is  the  focus  of  Cognitive  Work  /  Task 
Analysis  efforts. 
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•  Coherence  -  an  issue  of  the  visual  properties  of  the  representation.  Is  the  visual 
representation  (the  visualization)  allowing  the  human  to  effectively  perceive  and 
extract  the  information?  Coherence  addresses  the  question  of  how  the  various 
elements  within  a  visualization  compete  for  attentional  and  cognitive  resources  of 
the  perceiver.  For  example,  are  the  intended  distinctions  conveyed  by  the 
visualization  actually  discriminable  by  the  operator?  Do  the  elements  combine  to 
produce  the  desired  global  properties? 

Thus,  the  effectiveness  of  visualizations  is  determined  by  both  correspondence  and 
coherence.  More  specifically,  a  visualization’s  effectiveness  is  a  function  of  the  mapping 
between  the  work  domain,  the  presentation  elements  contained  in  it,  and  the 
extracted/perceived  information.  So,  a  CA  must  be  to  identify  relationships  between  the 
types  of  cognitive  work  requirements  (i.e.,  decisions)  being  supported  and  the  form  (e.g., 
visualization  technique)  for  presenting  the  information  to  support  those  demands.  The 
critical  property  is  how  the  underlying  cognitive  work  requirements  map  into  the 
structure  and  behavior  of  the  visual  elements.  This  has  several  critical  implications: 

•  The  need  for  an  approach  to  uncover  the  essential  cognitive  work  requirements  to 
be  supported  by  the  visualization; 

•  The  dynamic  nature  of  the  mapping  -  one  must  consider  how  the  display  behaves 
or  changes  as  the  state  of  the  problem  domain  changes; 

•  The  constraints  of  the  ‘virtual  perceptual  field’  offered  by  the  computer-based 
display  system  that  exacerbates  the  mapping  challenge. 

Our  ACWA  approach  explicitly  addresses  this  issue  by  maintaining  relationships 
between  the  types  of  cognitive  work  requirements  (i.e.,  decisions)  being  supported  and 
the  form  (e.g.,  visualization  technique)  for  presenting  the  information  to  support  those 
demands  through  the  specification  of  Representation  Design  Requirements.  These 
requirements  define  the  shaping  and  processing  for  how  the  information  /  relationships 
should  be  represented  to  practitioner(s);  in  essence,  a  formalization  of  Woods’  mapping 
principle. 

Conclusions 

Developing  effective  C2  decision  support  systems  is  a  significant  design  challenge  and 
requires  a  deft  design  hand  and  a  leading  edge  decision  centered  cognitive  analysis 
methodology  as  the  basis.  Key  to  the  redesign  challenge  is  the  highly  dynamic  and  open 
environment  in  which  the  system  must  operate,  and  consequently  the  varied  and 
unpredictable  demands  imposed  on  the  human  operators.  Undoubtedly,  the  system  must 
support  operators  in  following  established  principles  and  recommended  procedures,  but  it 
must  also  avoid  over-constraining  operators  and  avoid  hampering  them  from  taking 
advantage  of  their  abilities  to  improvise,  reason,  and  respond.  Thus,  the  major  focus  of 
this  type  of  effort  must  the  application  of  a  principled  and  coupled  methodology  geared 
towards  bridging  the  gap  between  analysis  and  the  design  of  decision  support  for  C2 
decision  making. 

Our  Applied  Cognitive  Work  Analysis  methodology  has  been  used  across  a  wide  range 
of  domains,  from  classically  designed  process  control  systems,  like  nuclear  power  plants, 
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to  so-called  ‘intentional’  domains,  such  as  military  command  and  control.  In  each  case,  it 
has  developed  decision  support  concepts  that,  in  hindsight,  appear  intuitively  obvious  (as 
an  ideal  decision  support  system  should),  and  yet  remained  undiscovered  prior  to  the 
application  of  ACWA.  In  ACWA,  decision  support  design  concepts  are  based  on  an 
explicit  analysis  and  modeling  of  the  work  that  is  to  be  supported.  Work  analysis  is 
therefore  not  an  end  in  itself  but  rather  a  means  to  derive  implications  for  design.  For 
tool-based  design  concepts  (e.g.,  a  computer-based  decision  support  system),  this  is 
valuable  to  the  extent  that  it  gives  designers  insights  into  how  to  create  tools  that 
effectively  support  human  work. 

Only  when  decision-centered  approaches  are  cost  effective,  directly  contribute  to  the 
value  of  the  end  decision  support  system,  and  have  the  reliability  and  credibility  of  an 
engineering  process  (rather  than  based  on  epiphanies)  will  cognitive  analysis  make  an 
effective  impact  on  the  systems  being  built  every  day. 
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the  ability  to  re-orient  focus  in  a  changing  world. 

Resilience  - 

the  ability  to  anticipate  and  adapt  to  surprise  and  error. 


CWA  Evaluation  Criteria 


Efficiency  ofthe  CWA  in  itseTT 

-■ Are-t-he-resources  4Deing  -invested-in-t-he 
CWA activities  comme nsurate  with-tfoe- 
value  of4be-resuJts-being  obtained?— 


Validity  of  the  CWA 

■  Does  it  capture  what  it  is  like  to  function  in 
the^ieid  of-pr-aetiee?— 


Effectiveness  of  CWA  in  design 

■  Does  the  CWA  point  to  what  is  likely  to  be 
useful  support? 

■  Does  it  help  generate  new  aiding  concepts 
and  innovations? 

■  Does  it  generate  ideas  that  can  be  readily 
converted  to  system  requirements  to  guide 
system  design  and  testing? 


-Tractabiiity  of-G  WA-resultsTn-d  esign  - 

~  AreThe  products" oT  the  CWA  documentedin  a 
way  that  can  be  meaningfully  reviewed, 
tracked,  and  updated  nofconly  throughout  the 
CWA  phase  but  also  throughout  the  entire 
system  design  life-cycle? 

■  Do  the  products  of  the  CWA  make  contact  with 
artifacts  utilized  in  the  software  design  process 
and  can -the  results  of -the  CWAbe  integrated 
into  the  software  and  product  development 
process? 


Predictive  Power  of  the  CWA 

■  Does  it  help  anticipate  the  impact  of  the 
introduction  of  new  technologies  and  aiding 
concepts  on  practitioner  performance? 

■  Does  it  predict  how  new  technological  power 
can  change  roles,  expertise  and  error? 

■  Does  it  help  address  the  envisioned  world 
problem? 
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Coordinating  Multiple  Perspectives  - 

Coordinating  shifts  and  contrasts  among  different  perspectives  over  task 
sequences  and  as  situations  evolve.  Coordination  of  multiple  points  of 
view  into  the  processes  of  the  work  domain  to  create  a  virtual  perceptual 
field  or  a  workspace  in  which  practitioners  carry  out  their  domain  activities. 


Making  Sense  of  Change  - 

Organize  and  integrate  pattern  views  around  classes  of  events  in  the  work 
domain  in  order  to  reveal  operationally  interesting  changes.  The  key  is  to 
transform  from  changes  in  individual  physical  characteristics  to  changes  of 
higher-order  semantic  properties. 


cogrtiT'iTf 
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Pattern  Views  - 

Organize  data  to  reveal  relationships  and  patterns  in  the  work  domain. 
Integrated  representations  inter-relate  data  to  provide  a  coherent  view  into 
a  process.  The  key  criterion  is  to  help  practitioners  pick  up  status  at  a 
glance  rather  than  having  to  read  and  mentally  integrate  many  individual 
pieces  of  data. 


DesjgnlTBcfjmques 


Events  —  Highlight  spatial  and  temporal 
events.  Representations  should  reveal  the 
dynamics,  evolution  and  future  paths  for  the 
process  in  question.  Events  are  temporally 
extended  behaviors  of  the  device  or  process 
involving  some  type  of  change  in  objects  or 
situations. 


Frames  of  Reference  - 

Discover  frames  of  reference  that 
capture  meaningful  relationships  in  the 
work  domain.  Each  frame  of  reference 
is  like  one  perspective  from  which  one 
views  or  extracts  meaning  from  data 
about  the  underlying  process  or  activity. 
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Contrast  —  Highlight  contrasts. 
Meaning  lies  in  contrasts  -  some  departure 
from  a  reference  or  expected  course. 
Representing  contrast  means  that  one 
indicates  the  relation  between  the  contrasting 
objects,  states  or  behaviors. 


(ogni 
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Context  —  Put  data  into  context;  the  context  of  related 
values  and  around  important  issues  in  the  work  domain.  One 
prerequisite  is  to  know  what  relationships  are  informative  in 
what  contexts  in  the  field  of  practice. 


The  ACWA  Methodology. 7?  Providing  an 
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CWA  Model: 

Modeling  Fundamental  Relationships 
and  Demands  in  the  Work  Domain 


Knowledge  Elicitation: 

Finding  demands,  decision-making 
challenges,  expert  strategies,  etc. 


I  nformation  Requirements: 

Defining  information  transfer 


Represen  ion  Requirements: 


requirements  for  effective 


decision- making 


dynamics  of  support  tools  to  achieve 


information  transfer  requirements 


Explicit  linking  to 
follow-on  design 
phase 


Mapping  Principle 


Cognitive  Work  Requirements: 

Defining  requirements  for  effective 
decision-making  in  work  domain 


Unfriendly,  2nd  Tier  General  of  Protection  Forces 


Information  Collection  Request  Set 


Collection  Value  Tradeoff 


Collection  Request  Details 


■  LfreUkDrai  TCT-f 
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<  <Unknowr>»  Ftequ  e$ting  Col  lection 


UnHwcf 
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Collection  Asset  Engagement 


Disrupted  Collections 


Collection  Asset  Precision  Level 

■  Acoustic  2 

■  UV  Collection  -5 
o'  UV  Collection  -6 

■  RF  Signal  - 1  © 

■  RF  Signal  -2  © 

■  UV  Collection  -3 

m  Infrared  -23  ® 

■  UV  Collection  -4 

a'  Visual  (High  Res)  -1  © 

■  Visual  (Low  Res)  -3 

■  Infrared  -44  © 

■  Visual  (Med  Res)  -4  © 

■  Visual  (High  Res)  -12  © 

■  Visual  (Low  Res)  -1 

m  Infrared  -8  © 

■  Acoustic  1 

m  Visual  (High  Res)  -8  © 

m  Visual  (Med  Res)  -2  © 

■  Infrared  -3  © 


Commitment  Horizon 


Radio  Freq. 


Infrared 
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Information  Collection  Request  Set 
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I  Unplanned  TCTs 
0  Unplanned  Routines 


Last  Time  of 
Collection 

□pp. 


Planned 

Collection 

Time 


Disrupted  TCTs 


Disrupted  Routines 


Potentially  Disrupted  Routines 
Planned  TCTs 
Planned  Routines 


Cognitive  Work  and  Info  Rqmts: 

rjC-Monitor-Col  I  ectio  n^Alm-status4A/itb — 
respect  to  the  collection  requests 

■  Achte  vemen  t-stat-u  s-ofindi  vidu  a\- 
aims  overtime. 

■  Commander's  priority  of  each  aim. 

■LStatus-of.  each-coJJection  j*equest_ 

_ (pJanned ,  jjcpJanned,  disrupted.) _ 

based  on  existence  of  associated 
Collection  Aim. 

■  Latest  time  of  collection  return 
value  for  each  individual  collection 
request  with  respect  to  the  set. 


Representation  Design  Rqmts: 


■ — DepiGt  each  result-as-independent- 
~symt3o1  of  equal^creen  re~al~estate.~ 

-■ — Encode  states  and-  priority-withi neach- 
symbolr  highei^lienceityrlTTglT-pn 
"TCTs. 


■  Spatially  separate  planned  and 
unplanned  results;  higher  salience  for 
unplanned. 


Depict  request  time  with  reference  to 
jcjgjpgt  and  lafesfJmeT 

Information  Collection  Request  Set 
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■  Unplanned  TCTS 
O  Unplanned  Rwrtlnes 


Disrupted  TCTs 
"  Dtmgjtpd  Routines 
>  Potent  iallv  Disrupted  Routines  I 
Planned  TCTS- 
Planned  Rwiines 


Cognitive  Work  and  Info  Rqmts: 

^^MonTtoF-Col  I  ecten^ATm-status4A/it^: — 
respect  to  the  collection  requests  “ 

- ■  Achie  veme  n  t-statu  s-efHnd-i  vidi  a! - 

aims  over  time. 

■  Commander's  priority  of  each  aim. 

KJL  Status  JDLeacbxoJJectioa  request 

_ {planned ,  jjnpJanned,  disrupted) _ 

based  on  existence  of  associated 


Representation  Design  Rqmts: 

-■ — Depict  each-r-esult-as-independent 
symbol  of  equal  screen  real  estate. 

-■ — Encode  status  and-  priority-withi n  each- 
syrrrbulr  highei^lienceiurlTTgfc 
Ids. 

■  Spatially  separate  planned  and 
unplanned  results;  higher  salience  for 
unplanned. 


Collection  Aim. 

■  Latest  time  of  collection  return 
value  for  each  individual  collection 
request  with  respect  to  the  set. 


Depict  request  time  with  reference  to 
T|||e|t  and  lafesfJmeT 

Information  Collection  Request  Set 
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■  Unplanned  TCTi 
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>  Potent  iallv  Disrupted  Routines  I 
Planned  TC15- 
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Cognitive  Work  and  Info  Rqmts: 


Monitor-Col  I  ectio  n^im-status-witb- 
respect  to  the  collection  requests 

Bteh-ie  vemen  t-staty  s-ef-indi  vidu  a]- 
aims  over  time. 

mmander's  priority  of  each  aim. 

jtus  of  each  collection  request 

_ ipJanned ,  jjnpJanned,  disrupted) _ 

based  on  existence  of  associated 
Collection  Aim. 

■  Latest  time  of  collection  return 
value  for  each  individual  collection 
request  with  respect  to  the  set. 


ACWA 


Functional 


Representation  Design  Rqmts 


■ — DepiGt  each  -result-as-independent- 
~syml3o1  of  equal^creen  re~al~estate.~ 

-■—Encode  statu  sand  -pr-iof-ity-within-eaeh 

-symooi*  fflgtrcrgaimiiFeT arjm  gn-ptinmy 


Spatially  separate  planned  and 
unplanned  results;  higher  salience  for 
unplanned. 

Depict  request  time  with  reference  to 
current  and  latest  time. 


Information  Collection  Request  Set 
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O  Unplanned  Rwrtl«ws 
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Cognitive  Work  and  Info  Rqmts:  Representation  Design  Rqmts: 


^JVIontor-CoJIeotjon^Ajm-stat^is-wit-h—  Depict  each  -r-esult-as-independent- 

respectto^the^llectionTTeqiiests  symbol  of  equal-screen  real" estate \  > 

BMHI^-ebie  vemen t-staty s-of-indi  vidu al - ■  Encode  status  and  priority-withi neaeh — 

aims  over  time.  symbol;  higher  salience  for  high-priority 


■  Commander's  priority  of  each  aim. 

Status-ofj  each-coJJection  jjeqo  est_ 

_ Ipteiaoed,  disrupted) _ 

based  on  existence  of  associated 
Collection  Aim. 


■  Latest  time  of  collection  return 
value  for  each  individual  collection 
request  with  respect  to  the  set. 


Cognitive  Work  and  Info  Rqmts: 

^jC-Monitor-Gol  I  ectio  plAfm-status4A/itb — 
respect  to  the  collection  requests 

■  Achte  vemen  t-statu  s-ofindi  vidu  at- 
aims  overtime. 

■  Commander's  priority  of  each  aim. 

JjLStatus-of.  each-coJJection  j*equest_ 

_ {pJamed ,  jjnpJanned,  disrupted.) _ 

based  on  existence  of  associated 
Collection  Aim. 


5  time  of  ooJJodcion  return 


value  for  each  individual  collection 
request  with  respect  to  the  set. 


Representation  Design  Rqmts: 


■ — DepiGt  each  -result-as-independent- 
"symfiol  of  equal^creen  re~al~estate.~ 

-■ — Encode  status  and-  priority-withi neach- 
symbolr  highei^lienceityrlTTglT-pn 
"TUTsT 


Spatially  separate  planned  and 
unplanned  results;  highe r  salience  for 
unplanned. 


Do  pi  of  request  time  with  reference  to 


■  Cogninve7^naIysis  mnsDae  10j£iQlGap.tIIcfi.lBB-  CQgo3Ive_ 
-  ^emands-oHtie-envisioned  ^-work  domaigajM  mmmmb 

- ■— AGWAHs-based-on-a-robust  framework  of-decision-makiag  in 

complex  work  environments. 


Gogflitive-Analysis-mtisHf-u  ly  Hfitegr-ate-witb-and - 

signifieanttyHmpacHE^-deve  I  opm  ent-  efforts;—— - — — — 

■  ACWA  extends  well  past  cognitive  analysis...  it  produces  design 
artifacts  based  on  Cognitive  Systems  Engineering  principles  that 
directly  support  Decision  Support  System  implementation; 

■  ACWA  specifies  the  mapping  from  the  demands  of  the  work 
domain  to  the  representation  for  the  warfighter. 


Cogiitiv^l^alysisJQMvl  make  a  difference  inJ 
by  subpofjnglhe  fesign  oil  innovative  Decision  Support, 


Systems! 
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■  £^oJlila^J^lEdTZHandbobkjDJ.  CogoiB-ve  IfasicDfisigK 
wnenEfT E0Banm7Z2D:D3~  ^1  SB  Nr^l-8D5S14DD3|6^~ 

■  Chap  T6T  Applied  Cognitive  Work  Analysis:  A~Fragmatic 

- Methodology - 

■  Chap  27:  Case  Studies:  CWA  in  the  design  of  Advanced 
Decision  Support 


